Paper 2: Implementation Challenges and Results 

Kirk Walters and Nicholas Sorensen, American Institutes for Research 
kwalters@air.org nsorensen@air. org 

Purpose / Objective / Research Question / Focus of Study^: 

This paper describes the implementation of the online and f2f summer algebra courses that were 
delivered in summers 2011 and 2012. These data will be used to frame the impact results 
presented in Paper 1. In particular, the paper will provide a detailed picture of how the online 
course was structured and the types of supports provided to students; compare the algebra 
content and course rigor between the online and f2f classes; and examine students’ perceptions 
of the online course between summers 2011 and 2012, which were starkly different in terms of 
the software glitches that interfered with the delivery of the course in 2012. 

The paper will be guided by the following research questions; 

1) How were the key components of the online algebra credit recovery course implemented 
during the summers of 201 1 and 2012, including specific technology challenges? 

2) How did the online and f2f Algebra IB courses compare in terms of content rigor and grading 
expectations? 

3) How did the instructional experience compare for students taking online and f2f summer 
credit recovery courses, and between students who took the online course in 201 1 and 2012? 

Data Collection and Analysis: 

We will draw from several different sources of implementation data that were collected in both 
conditions in both summer 2011 and 2012 to answer the paper’s research questions. These 
include in-person observations of the online and f2f classrooms, archival data generated from the 
online course, student and teacher surveys, online mentor logs, and course materials (syllabi, 
annotated tables of contents). Each of these is described more fully below: 

1) Observations of f2f and online classrooms. Each f2f and online class was observed once 
during each summer session. The observation instruments measured student engagement; 
teacher-student and mentor-student interactions; and the instructional and management 
behaviors of the mentors and f2f teachers. 

2) Aventa online course archival data. The Aventa archival course data include amount of 
time students’ logged in and participated in various components of the course; algebra topics 
covered and mastered; and interactions between online teachers and students between online 
teachers and mentors. Data that capture the frequency and duration of students’ logging into 
the course will be one of the sources used to portray the nature of online course technology 
problems that occurred in summer 2012. 

3) Data collector field notes. In addition to the archival data, we will use field notes to describe 
the nature and degree of the interruptions to the online course that occurred in summer 2012. 

4) Surveys. All teachers and students completed surveys at the end of each session. Surveys of 
students in the online and f2f courses measured students’ experiences and satisfaction with 
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each type of course and their overall perceptions of interactions, support, and expectations 
from their teachers and mentors. The surveys also included validated measures of academic 
demand, engagement, teacher academic attention, and support. The teacher surveys included 
questions about their experience and satisfaction with each type of course and the criteria 
they used to calculate grades in the credit recovery courses. 

5) Mentor logs. The mentors completed daily logs that measured how they spent their time, 
such as the extent to which they assisted students with mathematics content, managed student 
behavior, and the amount and type of communication and collaboration between online 
teachers. 

6) Course materials. The f2f teachers provided syllabi, annotated tables of contents and/or 
other materials at the end of each session. These materials included the algebra topics 
covered and amount of time spent on each topic, which will be used to compare the algebra 
topics in the Aventa online course. 

We will generate descriptive statistics from all of these sources to answer the study’s first two 
research questions; we will use the first four data sources to describe how the online course was 
implemented (RQ 1), and we will use the second (archival data), third (data collection field 
notes) and sixth (course materials) data sources to compare the rigor of the f2f and online courses 
(RQ 2). To compare the perceptions of students in the online course with (a) students in f2f 
summer algebra and (b) students taking the online course in Cohort 1, we will construct scales 
measuring students’ instructional experiences during the Algebra I credit recovery class using 
procedures established by the Consortium on Chicago School Research. To test whether 
students’ self-reported instructional experiences are significantly different by condition or 
cohort, we will employ the basic HLM model used in Study 1 for continuous outcomes. 

Findings / Results: 

The preliminary findings from year 1 (summer 2011) indicated that students were engaged, 
cooperative and attentive in both the online and f2f algebra classes, with no statistically 
significant differences between conditions. The online mentors reported spending the majority of 
their time (62%) on administrative tasks, 28% of their time teaching mathematics; and 9% of 
their time communicating with the online teacher or Aventa technical support. As mentioned in 
Paper 1 , the grades in the online course were lower than the f2f course, and analyses of the 
teachers’ grading criteria revealed that the online mentors placed a stronger emphasis on tests 
and quizzes than the f2f teachers. 

These findings will be expanded to include ongoing analyses of the summer 2012 
implementation data, which will be completed by January 2013, in time for the March 2013 
SREE conference. 

Conclusions: 

As mentioned in Paper 1, the Year 1 impact findings showed no significant differences in credit 
recovery rates or end-of-course assessment scores between students in the online and f2f classes, 
but students in the f2f courses had higher grades than students in the online classes. This paper 
will build upon this finding by describing what the grading policies were in the online and f2f 
classes in both summer cohorts. The paper will also include Aventa archival records and field 
notes to describe the magnitude of the problems encountered with the online course software. 
These data, along with the student survey data from 2012, will be used to compare the results 
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from 2011, which showed high student engagement overall and no differences between the 
online and f2f classrooms. All of the conclusions drawn from these implementation analyses will 
frame the interpretation of impact results. 
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